mAnI: Movie Amalgamation using Neural Imitation

Cross-modal data retrieval has been the basis of various creative tasks performed by Arti€cial Intelligence (AI). One such highly challenging task for AI is to convert a book into its corresponding movie, which most of the creative €lm makers do as of today. In this research, we take the €rst step towards it by visualizing the content of a book using its corresponding movie visuals. Given a set of sentences from a book or even a fan-€ction wriŠen in the same universe, we employ deep learning models to visualize the input by stitching together relevant frames from the movie. We studied and compared three di‚erent types of seŠing to match the book with the movie content: (i) Dialog model: using only the dialog from the movie, (ii) Visual model: using only the visual content from the movie, and (iii) Hybrid model: using the dialog and the visual content from the movie. Experiments on the publicly available MovieBook dataset shows the e‚ectiveness of the proposed models.